Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Try store response key as columns #9360

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Try store response key as columns #9360

wants to merge 2 commits into from

Conversation

yngve-sk
Copy link
Contributor

@yngve-sk yngve-sk commented Nov 27, 2024 •
edited
Loading

Seeing if it resolves some memory/timing issues wrt plotting. Should be faster wrt plotting since parquet is column-oriented, lets us do a select by response key without doing a scan.
Includes
cc53a75
PR:

============================= test session starts ==============================
collecting ... collected 5 items

test_plot_api.py::test_plot_api_big_summary_memory_usage[1-100-100-1200] 
test_plot_api.py::test_plot_api_big_summary_memory_usage[1000-100-100-1500] 
test_plot_api.py::test_plot_api_big_summary_memory_usage[2000-100-100-1950] 
test_plot_api.py::test_plot_api_big_summary_memory_usage[2-5803-11787-5500] 
test_plot_api.py::test_plot_api_big_summary_memory_usage[10-5803-11787-13500] 

================== 5 passed, 5 warnings in 304.53s (0:05:04) ===================
PASSED [ 20%]stats.total_memory_allocated / (1024 ** 2) = 790.8768510818481
stats.peak_memory_allocated / (1024 ** 2) = 777.6036682128906
PASSED [ 40%]stats.total_memory_allocated / (1024 ** 2) = 661.3918046951294
stats.peak_memory_allocated / (1024 ** 2) = 45.87281799316406
PASSED [ 60%]stats.total_memory_allocated / (1024 ** 2) = 1272.691068649292
stats.peak_memory_allocated / (1024 ** 2) = 56.69258975982666
PASSED [ 80%]stats.total_memory_allocated / (1024 ** 2) = 1915.6734647750854
stats.peak_memory_allocated / (1024 ** 2) = 294.8666820526123
PASSED [100%]stats.total_memory_allocated / (1024 ** 2) = 6075.969129562378
stats.peak_memory_allocated / (1024 ** 2) = 319.02239990234375

============================= test session starts ==============================
collecting ... collected 1 item

test_memory_usage.py::test_memory_smoothing 

====================== 1 passed, 10021 warnings in 2.59s =======================
PASSED                       [100%]
260.3829584121704

Main:

============================= test session starts ==============================
collecting ... collected 5 items

test_plot_api.py::test_plot_api_big_summary_memory_usage[1-100-100-1200] 
test_plot_api.py::test_plot_api_big_summary_memory_usage[1000-100-100-1500] 
test_plot_api.py::test_plot_api_big_summary_memory_usage[2000-100-100-1950] 
test_plot_api.py::test_plot_api_big_summary_memory_usage[2-5803-11787-5500] 
test_plot_api.py::test_plot_api_big_summary_memory_usage[10-5803-11787-13500] 

================== 5 passed, 5 warnings in 203.09s (0:03:23) ===================
PASSED [ 20%]stats.total_memory_allocated / (1024 ** 2) = 791.1395416259766
stats.peak_memory_allocated / (1024 ** 2) = 777.3985557556152
PASSED [ 40%]stats.total_memory_allocated / (1024 ** 2) = 1299.3754739761353
stats.peak_memory_allocated / (1024 ** 2) = 26.701663970947266
PASSED [ 60%]stats.total_memory_allocated / (1024 ** 2) = 2586.5789098739624
stats.peak_memory_allocated / (1024 ** 2) = 44.60098075866699
PASSED [ 80%]stats.total_memory_allocated / (1024 ** 2) = 4748.567058563232
stats.peak_memory_allocated / (1024 ** 2) = 3264.425220489502
PASSED [100%]stats.total_memory_allocated / (1024 ** 2) = 10482.615956306458
stats.peak_memory_allocated / (1024 ** 2) = 4480.5712032318115
   
 
test_memory_usage.py::test_memory_smoothing 

====================== 1 passed, 10021 warnings in 2.81s =======================
PASSED                       [100%]
283.1577425003052

Testing on memory_issue_when_plotting case:
Main:

- Time to open plotter: 10s
- Time to select FOPR: 4s
- Time to select FOPT: 4s
- Time to select FPR: 5.5
- Time to select WBHP:P-9P: 9s
- Manage experiments ...
- Open "observations" for ensemble: 17s

PR:

- Time to open plotter: 6.6s
- Time to select FOPR: 4s
- Time to select FOPT: 2.5s
- Time to select FPR, main: 3.26s
- Time to select WBHP:P-9P: 5.3s
- Manage experiments ...
- Time to migrate... 8to9: 1m48s
- Open "Observations". PR: 22.4s (diff is prob due to observations still being non-pivoted, can be optimized)

Running drogon_ahm update takes 3m26s with this, vs 3m9s on bleeding.

@yngve-sk yngve-sk force-pushed the 24.11.27.try-store-response-key-as-columns branch 4 times, most recently from 9f6c6b5 to e85661a Compare November 27, 2024 09:03
@yngve-sk yngve-sk self-assigned this Nov 27, 2024
@yngve-sk yngve-sk force-pushed the 24.11.27.try-store-response-key-as-columns branch 17 times, most recently from 98379ee to 2644ccd Compare November 27, 2024 14:55
@yngve-sk yngve-sk marked this pull request as ready for review November 27, 2024 15:04
@yngve-sk yngve-sk force-pushed the 24.11.27.try-store-response-key-as-columns branch 7 times, most recently from 50805c2 to 3501d80 Compare November 28, 2024 10:32
@yngve-sk yngve-sk force-pushed the 24.11.27.try-store-response-key-as-columns branch 4 times, most recently from 1aacc17 to bcb6e10 Compare November 28, 2024 12:33
@yngve-sk yngve-sk force-pushed the 24.11.27.try-store-response-key-as-columns branch from bcb6e10 to 9dbebcb Compare December 12, 2024 09:10
@yngve-sk yngve-sk mentioned this pull request Dec 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@yngve-sk yngve-sk

Labels
christmas-review
Projects
SCOUT
Status: Backlog
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yngve-sk @eivindjahren